Cooperative Understanding of Utterances and Gestures in a Dialogue-Based Problem Solving System

A dialogue plays an important role in learning how to solve a problem and form a concept. We are developing a problem solving and knowledge acquisition system based on co-reference between drill texts and dialogue with a teacher, focusing on first-grade mathematics. This paper presents a method of cooperative understanding of utterances and gestures within dialogue. We first describe our system design principles, which provide the basis for the integration of multimodal information during a dialogue. We define a principle of complementarity, explain its implementation, and describe the architecture of the problem solving system. We then show how to integrate our algorithms for utterance and gesture analysis within that software architecture. A feature-based approach is used for gesture recognition, derived from a sequence of images arising during the cooperative analysis of utterances. We conclude with an evaluation of the system against the design principles.